S9 Uncertainty Results
Introduction
This document presents the findings of various methods used to quantify the uncertainty of policy scenarios from the Dcision Support tool for Child and Adolescence Obesity, a system dynamics model developed to explore underlying relationships contributing to youth obesity.
Previous sensitivity analysis revealed an interaction between base model assumptions and the estimated change in BMI prevalence for each policy outcome. This document aims to compare uncertainty estimates from four different methods of sampling model input against initial conditions:
- 1 No parameter sampling: The parameters describing both the model and scenarios are maintained at their initial input values.
- 2 Policy scenario input assumptions: Parameters defining the implementation or effectiveness characteristics of the five intervention policy scenarios considered in the models are sampled.
- 3 Base model input assumptions: Parameters describing the initial conditions and numerical input values for the underlying model of childhood and adolescent overweight and obesity are sampled.
- 4 A combination of both model and policy input assumptions: Both sets of parameters are sampled to explore the interaction between model and policy assumptions.
Methods
A sensitivity analysis of 649 base model input assumptions was conducted, assessing changes in each of these assumptions for each age-gender-BMI outcome and the change in BMI from the policy scenario. The results of that analysis suggested that policy outcomes interact with base model assumptions. Among these 649 input variables, it was found that variables related to water and moisture consumption had zero impact on the outcomes due to the zero energy density of water. Additionally, variables concerning adult input assumptions were not included due to their minimal impact on childhood outcomes.
The median, along with the 2.5th and 97.5th quantiles, is employed to depict scenario uncertainty and communicate the effectiveness of the results
Sampling
The base model input assumptions were sampled 10,000 times probabilistically within the known 95% confidence interval (CI) range or ±10% of the model’s initial assumption in situations where confidence intervals were unavailable. Input assumptions were sampled symmetrically, using a normal distribution to ensure that the majority of the input assumption was centered on the initial assumption. Parameter input ranges were treated as a proxy for their 95% CI. Appendix tables in the thesis show the sampling ranges for sensitivity and uncertainty in Sections S1-S8. Latin hypercube sampling was employed to ensure a sufficient spread of input assumptions to represent the parameter state space. Input assumptions were assumed to be independent, and no covariance between input assumptions was incorporated into the sampling protocol. This sample was retained for each of the combined model and policy input assumption analyses.
For the analysis where only scenario input assumptions are being sampled, 1,000 assumptions are sampled to reduce computational costs. This sample uses the same method as described above: symmetric sampling using Latin hypercube sampling from parameter ranges that were either reported or measured 95% CI or ±10% of the initial estimate.
Base model input samples were repeated for each scenario with 10,000 randomly sampled scenario input assumptions. Ideally, factorial sampling would have been used; however, the computational cost was much too high. A factorial analysis would have randomly sampled scenario input assumptions for each of the 10,000 sampled base model input samples, creating 10,000^1000 (assuming 10,000 base and 1000 scenario samples) samples.
Dipiction of sampled parameters
Model overview
Figure 1: Model overview
Base model uncertainty
Figure 2 shows the energy balance model component with the tested sources of model uncertainty circled in red. These parameters describe characteristics and definitions about the model. For example, the model development choices to aggregate discretionary foods according to the AUSNUT database impact the levels of consumption and the proportion of nutrients within each food group. If the food items included in the definitions of discretionary food are different from those chosen for the model, then these values would change. Numerical uncertainty is applied to each of these circled variables through sampling, which then propagates throughout the model, impacting the model outcomes. Many of these variables are stratified by age-gender-BMI groups.
Figure 2: Enegy balance component with circles representing sources of uncertainty
Figure 3 illustrates the structure that influences how children enter the model. The associations from the individual-level analysis of combined early prevention of obesity trials are varied to reflect the uncertainty associated with these regression coefficients.
Figure 3: Intergenerational structure
Policy scenario uncertainty
Along with the initial values of the base model, inputs that describe each policy scenario (Figure 4) were also sampled to portray the uncertainty of these estimates. The interaction between these variables and the sampled base model inputs is measured in the results.
Figure 4: X
Relapse was an important concept in modeling the long-term effectiveness of each policy. The structure used to model relapse is presented in Figure 5, with the red circles indicating where the relapse rate was sampled.
Figure 5: X
Results
Initial base model input values with initial scenarios input values
In these results, the initial input conditions for the base model and policy scenarios are applied to the model. Figure 6 shows the reduction in obesity for children and adolescent. This plot compares the results of each intervention policy to the base model, where no policy in implemented.
The reduction in overweight for each aggregate age group is presented in the appendix.
Figure 6: Initial inputs
Initial base model input values with sampled scenarios input values
This section presents the model findings when the scenario input assumptions were randomly sampled. The plots below show the change in BMI from the model with no scenario being implemented for each of the 1,000 samples.
Using the uncertainty range presented in the thesis appendix, the policy scenario input variables were randomized to assess the impact of uncertainty on policy effectiveness. Figure 7 presents the median and 95% inner percentiles (2.5th and 97.5th). This result suggests that there is little change to the model estimates and relatively low uncertainty with sampled scenario inputs. Furthermore, the distribution of each scenario is symmetric, with similar distances from the lower percentile to the median and from the median to the upper percentile.
The reduction in overweight for each aggregate age group is presented in the appendix.
Figure 7: Sampled scenario inputs
Sampled base model input values with initial scenarios input values
Figure 8 shows the results from the uncertainty analysis when base model input assumptions were randomly sampled, and the initial input conditions for each policy scenario variable were applied to the model. These results have much wider inner 95% percentile ranges, and the median point estimate differs, likely caused by a difference in the distribution of policy effectiveness (shown in the following results sections).
These results portray a higher level of uncertainty for the modelled effectiveness of each scenario, compared to when only scenario input parameters were sampled. This is logically consistent and aligns with findings in the sensitivity analysis, where the interaction between base model and scenarios had a significant impact on the modelled policy effectiveness.
The reduction in overweight for each aggregate age group is presented in the appendix.
Figure 8: Sampled base model input values
Sampled base model input values with sampled scenarios input values
The results in Figure 9 show the modelled policy effectiveness when both sets (base model and scenario) input parameters are sampled. These results replicate the previous analysis where only base model parameters were sampled. This suggests that the small variability seen in the scenario-only analysis provides minimal impact on the results’ outcome.
Figure 9: Sampled base model and sampled scenario input values
Comparison of outcome distributions
In this section, the four methods of estimating the effectiveness of each scenario are compared by plotting the distribution of estimated reduction in overweight and obesity over time. These results show small differences in the results between methods. However, in the scenario with SSB, CCI, and EPOCH + CCI, we see that when the base model inputs are sampled, the effectiveness is slightly lower than when initial inputs are used or in the scenario-sampled analysis. This is caused by a higher number of model results closer to zero effectiveness, dragging the estimate down consistently over the modeled time.
Children
Figure 10: Change in childhood obesity comparing sampling methods overlayed
Figure 11 shows the distributions for each individual policy effectiveness at the end of the model (2057). Each of the panel rows represents a change in sampling strategy. The blue solid line represents the median of the distribution, with blue dashed lines indicating the 2.5th and 97.25th percentiles, respectively. This plot shows the dramatic increase in mode uncertainty when the base model inputs are included.
Means and 95%CI are calculated and presented to compare the communication of model scenario.
Due to the possibility of skewed data, means can misrepresent the communication of results’ central tendency. Also, it is important to note that 95% CI represents the certainty of the mean value and does not reflect the certainty of the model results. This means that the interpretation of 95% CI doesn’t align with model uncertainty but with mean uncertainty. As a result, 95% CI is dependent on and sensitive to run size; a higher number of runs will increase the certainty of what the mean should be. However, since we can sample runs indefinitely, this also misrepresents model certainty by artificially deflating the range. For these reasons, means and 95% CI are not appropriate measures to represent modelled effectiveness or model uncertainty.
Figure 11: Change in childhood obesity comparing distributions
Adolescents
Figure 12: Change in adolescent obesity comparing sampling methods overlayed
Figure 13: Change in adolescent obesity comparing distributions
Synergy
Synergy was defined as the interaction of two or more policies so that their combined effectiveness is estimated to be greater than the sum of the individual effectiveness. The table below uses the median effect of the combined policy scenario, with the corresponding individual policy effectiveness. The corresponding sum of individual effects can be compared with the combined policy scenario to estimate a ratio of combined policy divided by the sum of individual. A synergy ratio greater than one suggests that when policies are implemented together, there is a greater effect than the individual effectiveness would suggest. Similarly, a ratio less than one suggests an antagonistic effect, and that there is a loss in effectiveness.
| Age group | Policy | Median | EPOCH | CCI | SI | SVI | SSB | Sum of individual | Synergy ratio |
|---|---|---|---|---|---|---|---|---|---|
| Children | EPOCH+CCI | -0.2420982 | -0.0334217 | -0.1882334 | NA | NA | NA | -0.2216551 | 1.0922291 |
| Children | SI+SVI | -0.5634360 | NA | NA | -0.1461693 | -0.4188805 | NA | -0.5650499 | 0.9971438 |
| Children | EPOCH+CCI+SI+SVI | -0.8348312 | -0.0334217 | -0.1882334 | -0.1461693 | -0.4188805 | NA | -0.7867050 | 1.0611743 |
| Children | EPOCH+CCI+SI+SVI+SSB | -1.4331298 | -0.0334217 | -0.1882334 | -0.1461693 | -0.4188805 | -0.5730093 | -1.3597144 | 1.0539932 |
| Adolescents | EPOCH+CCI | -0.0769366 | -0.0079550 | -0.0530346 | NA | NA | NA | -0.0609896 | 1.2614694 |
| Adolescents | SI+SVI | -1.0366960 | NA | NA | -0.3995330 | -0.6310581 | NA | -1.0305911 | 1.0059236 |
| Adolescents | EPOCH+CCI+SI+SVI | -1.1439887 | -0.0079550 | -0.0530346 | -0.3995330 | -0.6310581 | NA | -1.0915807 | 1.0480111 |
| Adolescents | EPOCH+CCI+SI+SVI+SSB | -1.8144023 | -0.0079550 | -0.0530346 | -0.3995330 | -0.6310581 | -0.6093323 | -1.7009130 | 1.0667226 |
Final results
The final results consider the widest and most conservative approach to representing policy effectiveness. This was observed when both sets of input parameters were randomly sampled. Additionally, the median and inner 95 percentiles (2.5th and 97.5th percentiles) best reflect model certainty. It should be noted that since calibrated inputs were not adjusted for each sample, the uncertainty is likely higher (and again more conservative) in these results.
While a conservative approach was preferred, the modeled uncertainty is reasonable; the widest interval was in the reduction in obesity in adolescents when all interventions (a-e) were implemented. This result produced a 4.28 pp wide interval (0.15-4.43).
Figure 14: Forrest plot, reduction of obesity by senarios and age group
Figure 15: Change in obesity overtime, by scenario and aggregate age groups
Conclusion
This report compares different sampling methods and how the inclusion of different variables impacts model certainty when calculating the effectiveness of each policy scenario. The results show that certainty of policy effectiveness is conditional on the base model input values and that a conservative approach is to consider both the parameter input uncertainty in the model and sampled values. The medians and 95% inner percentiles are a fair representation of model uncertainty, as 95% CI only reflect certainty of the mean, not of the model. A conservative approach was taken in the final results with a reasonable level of model uncertainty observed.
Appendix
Additional plots
Tables of effects
Children
Overweight| Scenario | Intial assumptions | Sampled scenario assumptions | Sampled Base assumptions | Fully sampled assumptions |
|---|---|---|---|---|
| EPOCH | -0.21 | -0.21 (-0.26, -0.15) | -0.05 (-0.61, 0.55) | -0.05 (-0.62, 0.55) |
| CCI | -0.53 | -0.52 (-0.65, -0.43) | -0.26 (-0.92, 0.12) | -0.26 (-0.92, 0.12) |
| SI | -0.55 | -0.55 (-0.69, -0.45) | -0.25 (-1.01, 0.21) | -0.24 (-1.02, 0.21) |
| SVI | -1.42 | -1.41 (-1.67, -1.20) | -0.59 (-2.26, 0.36) | -0.58 (-2.31, 0.35) |
| SSB | -1.37 | -1.36 (-1.57, -1.21) | -0.73 (-2.79, 0.38) | -0.73 (-2.82, 0.38) |
| EPOCH+CCI | -0.73 | -0.73 (-0.87, -0.61) | -0.32 (-1.27, 0.29) | -0.32 (-1.28, 0.31) |
| SI+SVI | -2.08 | -2.08 (-2.35, -1.82) | -0.81 (-3.18, 0.53) | -0.81 (-3.23, 0.54) |
| EPOCH+CCI+SI+SVI | -2.61 | -2.62 (-3.03, -2.29) | -1.15 (-4.18, 0.57) | -1.14 (-4.17, 0.56) |
| EPOCH+CCI+SI+SVI+SSB | -4.55 | -4.54 (-5.01, -4.13) | -1.83 (-6.77, 0.94) | -1.83 (-6.84, 0.94) |
| Scenario | Intial assumptions | Sampled scenario assumptions | Sampled Base assumptions | Fully sampled assumptions |
|---|---|---|---|---|
| EPOCH | -0.01 | -0.01 (-0.02, -0.01) | -0.03 (-0.58, 0.35) | -0.03 (-0.58, 0.38) |
| CCI | -0.38 | -0.38 (-0.47, -0.31) | -0.19 (-0.54, 0.00) | -0.19 (-0.54, 0.00) |
| SI | -0.06 | -0.06 (-0.06, -0.05) | -0.15 (-0.43, 0.00) | -0.15 (-0.44, 0.00) |
| SVI | -0.50 | -0.50 (-0.60, -0.40) | -0.42 (-1.07, 0.00) | -0.42 (-1.10, 0.00) |
| SSB | -0.93 | -0.93 (-1.09, -0.79) | -0.58 (-1.45, -0.02) | -0.57 (-1.49, -0.02) |
| EPOCH+CCI | -0.40 | -0.39 (-0.48, -0.33) | -0.25 (-0.91, 0.16) | -0.24 (-0.92, 0.18) |
| SI+SVI | -0.53 | -0.53 (-0.63, -0.43) | -0.57 (-1.43, 0.00) | -0.56 (-1.45, 0.00) |
| EPOCH+CCI+SI+SVI | -0.92 | -0.91 (-1.00, -0.81) | -0.84 (-2.06, -0.02) | -0.83 (-2.08, -0.02) |
| EPOCH+CCI+SI+SVI+SSB | -1.48 | -1.48 (-1.59, -1.38) | -1.43 (-3.44, -0.09) | -1.43 (-3.43, -0.09) |
Adolescents
Overweight| Scenario | Intial assumptions | Sampled scenario assumptions | Sampled Base assumptions | Fully sampled assumptions |
|---|---|---|---|---|
| EPOCH | -0.13 | -0.13 (-0.17, -0.10) | -0.00 (-0.35, 0.45) | -0.00 (-0.36, 0.45) |
| CCI | -0.17 | -0.17 (-0.23, -0.13) | -0.00 (-0.25, 0.30) | -0.00 (-0.25, 0.29) |
| SI | -0.85 | -0.85 (-1.07, -0.70) | -0.49 (-1.95, 0.32) | -0.49 (-1.94, 0.31) |
| SVI | -1.28 | -1.27 (-1.59, -1.01) | -0.63 (-2.83, 0.51) | -0.62 (-2.85, 0.50) |
| SSB | -1.06 | -1.07 (-1.33, -0.92) | -0.52 (-2.45, 0.52) | -0.51 (-2.48, 0.52) |
| EPOCH+CCI | -0.30 | -0.30 (-0.37, -0.25) | 0.01 (-0.49, 0.48) | 0.01 (-0.51, 0.48) |
| SI+SVI | -2.35 | -2.36 (-2.84, -1.97) | -1.11 (-4.80, 0.81) | -1.09 (-4.79, 0.82) |
| EPOCH+CCI+SI+SVI | -2.61 | -2.63 (-3.13, -2.19) | -1.12 (-5.03, 0.87) | -1.11 (-5.04, 0.88) |
| EPOCH+CCI+SI+SVI+SSB | -4.43 | -4.43 (-5.09, -3.85) | -1.61 (-7.47, 1.40) | -1.62 (-7.48, 1.38) |
| Scenario | Intial assumptions | Sampled scenario assumptions | Sampled Base assumptions | Fully sampled assumptions |
|---|---|---|---|---|
| EPOCH | 0.01 | 0.01 (0.00, 0.01) | -0.01 (-0.40, 0.38) | -0.01 (-0.40, 0.40) |
| CCI | -0.09 | -0.08 (-0.11, -0.06) | -0.05 (-0.38, 0.08) | -0.05 (-0.38, 0.08) |
| SI | -0.46 | -0.46 (-0.53, -0.38) | -0.41 (-1.06, -0.00) | -0.40 (-1.05, -0.00) |
| SVI | -0.74 | -0.74 (-0.88, -0.62) | -0.63 (-1.60, -0.01) | -0.63 (-1.60, -0.01) |
| SSB | -0.77 | -0.76 (-0.90, -0.64) | -0.61 (-1.58, -0.02) | -0.61 (-1.62, -0.02) |
| EPOCH+CCI | -0.08 | -0.08 (-0.11, -0.06) | -0.08 (-0.63, 0.28) | -0.08 (-0.63, 0.28) |
| SI+SVI | -1.12 | -1.11 (-1.25, -0.96) | -1.04 (-2.58, -0.04) | -1.04 (-2.60, -0.04) |
| EPOCH+CCI+SI+SVI | -1.19 | -1.19 (-1.33, -1.04) | -1.15 (-2.91, -0.06) | -1.14 (-2.90, -0.06) |
| EPOCH+CCI+SI+SVI+SSB | -1.57 | -1.57 (-1.73, -1.44) | -1.81 (-4.45, -0.15) | -1.81 (-4.43, -0.15) |
Overweight plots
Figure 16: Initial inputs; overweight results
Figure 17: Sampled scenario inputs, overweight
Figure 18: Sampled base model input values, overweight
Figure 19: Sampled base model and sampled scenario input values, overweight
Figure 20: Change in childhood overweight comparing distributions
Figure 21: Change in adolescent overweight comparing distributions
Figure 22: Forrest plot, reduction of overweight by senarios and age group
Figure 23: Change in overweight overtime, by scenario and aggregate age groups